Split Samples and Design Sensitivity in Observational Studies

Ruth Heller, Paul R. Rosenbaum and Dylan S. Small University of Pennsylvania, Philadelphia Abstract. An observational or nonrandomized study of treatment e¤ects may be biased by failure to control for some relevant covariate that was not measured. The design of an observational study is known to strongly a¤ect its sensitivity to biases from covariates that were not observed. For instance, the choice of an outcome to study, or the decision to combine several outcomes in a test for coherence can materially a¤ect the sensitivity to unobserved biases. Decisions that shape the design are, therefore, critically important, but they are also di¢ cult decisions to make in the absence of data. We consider the possibility of randomly splitting the data from an observational study into a smaller planning sample and a larger analysis sample, where the planning sample is used to guide decisions about design. After reviewing the concept of design sensitivity, we evaluate sample splitting in theory, by numerical computation, and by simulation, comparing it to several methods that use all of the data. Sample splitting is remarkably e¤ective, much more so in observational studies than in randomized experiments: splitting 1000 matched pairs into 100 planning pairs and 900 analysis pairs often materially improves the design sensitivity. An example from genetic toxicology is used to illustrate the method.